feat(proof): stub-win sim scores against sealed baseline - #232
Conversation
Emit harness metrics relative to the sealed vector when PROOF_FORCE_SIM and PROOF_SIM_STUB_WIN are set, so staging submits can reach awaiting_admin. Skill-only sim stays NLL>=1.0. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Prefer 159.223.159.205/challenge/proof (ready sim) over the stale staging.api Lium instance. Submit both open topic ids. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live 201 on dt-no-ib-v0 and muon-vs-adamw-10m-v0 against 159.223.159.205; default sim misses the ~0.29 NLL floor. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Option A: Sim + sealed baseline always uses sim_win_document. No extra host env. Do not reseal (option B is ops-owned). Co-authored-by: Mathis <echobt@users.noreply.github.com>
Skill 0.95 (StubScorer::win) still NLL>=1.0 vs a 0.29 seal. Option B reseal is paused; do not reseal from this lane. Co-authored-by: Mathis <echobt@users.noreply.github.com>
c5c581a
into
cursor/proof-payout-snapshot-flops-ed26
Greptile SummaryThis change adds forced-Sim proof evaluation, submission validation coverage, and an operational submission probe. Two reproduced safety issues in Confidence Score: 1/5Not safe to merge until the operational probe restricts its destinations to approved environments and stops using predictable shared temporary files. Both reported failures were reproduced with the actual script against isolated loopback services, including observed mutating requests and observed symlink write-through. Files Needing Attention:
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| refuse_prod() { | ||
| local url="${1:-}" | ||
| if echo "$url" | grep -Eq "$PROD_HOSTS"; then | ||
| RED "refusing production host: $url" | ||
| exit 2 | ||
| fi |
There was a problem hiding this comment.
If a production service is addressed through its IP address or an alternate DNS name, this guard allows the probe because it only rejects URL strings containing two literal hostnames. The script then sends mutating submission POST requests to that origin, so an operator can unintentionally write test submissions to production. Require an explicit allowlist of approved local or staging origins and fail closed for every other destination. This must be addressed before merging.
How this was verified: A loopback IP URL passed the guard and received both submission POSTs, while a URL containing the denied hostname was rejected.
Knowledge Base Used:
Artifacts
- Authored executable harness that starts only loopback mock services and invokes the supplied script against denied-host, IP-literal, and redirect cases, ending with assertions about observed requests.
- Captured command output showing the implemented substring denylist and the guarded proof POST call sites before runtime validation, establishing the code under test.
- Captured local harness execution showing hostname rejection, IP-literal mutation requests to the loopback mock, and no request to the redirect target, confirming the alias/IP bypass but disproving redirect following.
| LOG "GET /v1/proof/topics → $(echo "$topics" | head -c 400)…" | ||
| echo "$topics" | grep -q 'content_sha256' && { RED "topics leaked holdout records"; return 1; } | ||
|
|
||
| code="$(curl -sS -m 8 -o /tmp/proof-e2e-empty.json -w '%{http_code}' \ |
There was a problem hiding this comment.
The probe writes responses to predictable filenames in shared /tmp locations at this call site and the equivalent calls later in the script. A local user can pre-create one as a symlink; curl -o follows it and overwrites the linked operator-writable file when the script runs. Create a private directory with mktemp -d, store all responses there, and remove it with an exit trap. This must be addressed before merging.
How this was verified: Pre-created symlinks for all three response paths remained symlinks and each linked target was overwritten by the local response body.
Knowledge Base Used: Deployment automation and infrastructure
Artifacts
- The authored executable starts only loopback HTTP servers, runs the supplied script in before and symlink-precreated modes, and inspects targets; it is the executed source.
- The command capture records the complete local harness source that was executed, including its loopback-only server setup and symlink assertions; it provides reproducible test input.
- The loopback-only before run completed successfully and created ordinary response files at the three script-selected paths; it establishes normal behavior.
- The loopback-only after run completed successfully while all three response paths remained symlinks and every linked target was overwritten with the response body; this confirms the vulnerability.
- The captured source search identifies all three fixed curl output paths and finds no mktemp, trap, or removal command in the script; it confirms affected locations and missing cleanup.
…235) Reland #232 onto current main. Under PROOF_FORCE_SIM, a sealed topic scores with sim_win_document (harness relative to the seal) so staging submit→score can clear a ~0.29 NLL floor. Skill-only sim_document stays NLL>=1.0. Lium path unchanged. Conflict resolution vs #234: keep ProxyModelMissing/HoldoutStoreMissing and live-asset notes; keep sim probe runbook. Cargo.lock regenerated incrementally for new test deps only. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mathis <echobt@users.noreply.github.com>
2 participants
Summary
Option A (this PR, lasting): under
PROOF_FORCE_SIM(eval_backend=sim), a sealed topic scores with harness numbers relative to the sealed vector (sim_win_document):epsilon_topic_max_regresstokens_per_sec≥ ref × (1 +epsilon_rel) when that is the primarySkill-only
sim_documentstays NLL ≥ 1.0 and cannot beat a real ~0.29 seal.StubScorer::winskill=0.95 also fails that floor. The Lium path never uses this helper.PROOF_SIM_STUB_WINis a leftover no-op — no extra host env.Option B (Développeur, paused): reseal staging baselines to
BASELINE_SKILL=0.40(NLL ≈ 2.94), resign topics, retest submit→awaiting_adminon staging only. Paused — Mathis redirected that lane to prod RLM E2E (1× GPU). This lane does not reseal or edit staging host files.Not production. No
set_weights. No Lium rent. No secrets. Do not merge as part of the E2E window.Live staging 2026-09-07 (pre-A binary)
Origin:
http://159.223.159.205/challenge/proof(eval_backend=sim,force_sim=true,can_score=true,baseline_sealed=true, offeropenrouter-glm53flash-v0open).staging.api.joinbase.aistill answers a stale Lium / fail-closed instance — do not POST there.topic_id is requirednot-a-real-topicunknown topicdt-no-ib-v0pf_0000000000000002rejectedmuon-vs-adamw-10m-v0pf_0000000000000003rejectedReceipts:
"provider":"sim". After this PR is deployed on that host (force_sim already on),--probeshould reachawaiting_adminwithout a reseal.Tests
cargo test -p proof-eval stub_win— skill=0.95 and skill=1.0 still NLL≥1.0 vs 0.29 seal;sim_win_documentmeets the three inequalities and passes judge; Lium ignores the helpercargo test -p proof-eval sim_plus_sealed_uses_relative_harness—eval_after_freezeSim+sealed uses the relative harnesscargo test -p proof-http sim_stub_win_submit_reaches_awaiting_admin— tight 0.29 seal →awaiting_admin/eligible=trueunder Simcargo test -p proof-http --lib— submit contract + fail-closed 400/503cargo test -p proof-challenge-bin --test submit_e2e—--force-simbinary, both topic idsRunbook:
docs/runbooks/proof-submit-e2e.mdGreptile
Every PR is reviewed by Greptile before merge. Config:
.greptile/.@greptileai reviewTest plan
cargo test -p proof-eval --libcargo test -p proof-http --libcargo test -p proof-challenge-bin --test submit_e2ecargo clippyon touched cratescargo test --workspacenot runRisk
Staging/dev only. Live Lium scoring unchanged. Droplet compose overlays stay sim-off.
Naming
I did not rename
BASE_*environment variables, deployed host paths, GHCR package names, orbase-*-v1domain tags.